Linear least squares (mathematics)

In statistics and mathematics, linear least squares is an approach to fitting a mathematical or statistical model to data in cases where the idealized value provided by the model for any data point is expressed linearly in terms of the unknown parameters of the model. The resulting fitted model can be used to summarize the data, to predict unobserved values from the same system, and to understand the mechanisms that may underlie the system.

Mathematically, linear least squares is the problem of approximately solving an overdetermined system of linear equations, where the best approximation is defined as that which minimizes the sum of squared differences between the data values and their corresponding modeled values. The approach is called "linear" least squares since the solution depends linearly on the data. Linear least squares problems are convex and have a closed-form solution that is unique, provided that the number of data-points used for fitting equals or exceeds the number of unknown parameters, except in special degenerate situations. In contrast, non-linear least squares problems generally must be solved by an iterative procedure, and the problems can be non-convex with multiple optima for the objective function.

In statistics, linear least squares problems correspond to a particularly important type of statistical model called linear regression which arises as a particular form of regression analysis. One basic form of such a model is an ordinary least squares model. The present article concentrates on the mathematical aspects of linear least squares problems, with discussion of the formulation and interpretation of statistical regression models, and of statistical inferences related to these are dealt with in the articles just mentioned. See outline of regression analysis for an outline of the topic.

Motivational example

The general problem

Consider an overdetermined system

$\sum_{j=1}^{n} X_{ij}\beta_j = y_i,\ (i=1, 2, \dots, m),$

of m linear equations in n unknown coefficients, β₁,β₂,…,β_n, with m > n. This can be written in matrix form as

$\mathbf {X} \boldsymbol {\beta} = \mathbf {y},$

where

$\mathbf {X}=\begin{pmatrix} X_{11} & X_{12} & \cdots & X_{1n} \\ X_{21} & X_{22} & \cdots & X_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ X_{m1} & X_{m2} & \cdots & X_{mn} \end{pmatrix} , \qquad \boldsymbol \beta = \begin{pmatrix} \beta_1 \\ \beta_2 \\ \vdots \\ \beta_n \end{pmatrix} , \qquad \mathbf y = \begin{pmatrix} y_1 \\ y_2 \\ \vdots \\ y_m \end{pmatrix}.$

Such a system usually has no solution, so the goal is instead to find the coefficients β which fit the equations "best," in the sense of solving the quadratic minimization problem

$\hat{\boldsymbol{\beta}} = \underset{\boldsymbol{\beta}}{\operatorname{arg\,min}}\,S(\boldsymbol{\beta}),$

where the objective function S is given by

$S(\boldsymbol{\beta}) = \sum_{i=1}^{m}\bigl| y_i - \sum_{j=1}^{n} X_{ij}\beta_j\bigr|^2 = \bigl\|\mathbf y - \mathbf X \boldsymbol \beta \bigr\|^2.$

A justification for choosing this criterion is given in properties below. This minimization problem has a unique solution, provided that the n columns of the matrix X are linearly independent, given by solving the normal equations

$(\mathbf X^\top \mathbf X )\hat{\boldsymbol{\beta}}= \mathbf X^\top \mathbf y.$

Derivation of the normal equations

Define the $i$ th residual to be

$r_i= y_i - \sum_{j=1}^{n} X_{ij}\beta_j$ .

Then $S$ can be rewritten

$S = \sum_{i=1}^m r_i^2.$

S is minimized when its gradient vector is zero. (This follows by definition: if the gradient vector is not zero, there is a direction in which we can move to minimize it further - see maxima and minima.) The elements of the gradient vector are the partial derivatives of S with respect to the parameters:

$\frac{\partial S}{\partial \beta_j}=2\sum_{i = 1}^m r_i\frac{\partial r_i}{\partial \beta_j} \ (j=1,2,\dots, n).$

The derivatives are

$\frac{\partial r_i}{\partial \beta_j}=-X_{ij}.$

Substitution of the expressions for the residuals and the derivatives into the gradient equations gives

$\frac{\partial S}{\partial \beta_j} = 2\sum_{i=1}^{m} \left( y_i-\sum_{k=1}^{n} X_{ik}\beta_k \right) (-X_{ij})\ (j=1,2,\dots, n).$

Thus if $\hat \beta$ minimizes S, we have

$2\sum_{i=1}^{m} \left( y_i-\sum_{k=1}^{n} X_{ik}\hat \beta_k \right) (-X_{ij}) = 0\ (j=1,2,\dots, n).$

Upon rearrangement, we obtain the normal equations:

$\sum_{i=1}^{m}\sum_{k=1}^{n} X_{ij}X_{ik}\hat \beta_k=\sum_{i=1}^{m} X_{ij}y_i\ (j=1,2,\dots, n).$

The normal equations are written in matrix notation as

$(\mathbf X^\top \mathbf X) \hat{\boldsymbol{\beta}} = \mathbf X^\top \mathbf y .$ (where X^T is the matrix transpose of X)

The solution of the normal equations yields the vector $\hat{\boldsymbol{\beta}}$ of the optimal parameter values.

Computation

A general approach to the least squares problem $\operatorname{\,min} \, \big\|\mathbf y - X \boldsymbol \beta \big\|^2$ can be described as follows. Suppose that we can find a n by m matrix S such that XS is an orthogonal projection onto the image of X. Then a solution to our minimization problem is given by

$\boldsymbol \beta = S \mathbf y$

simply because

$X \boldsymbol \beta = X ( S \mathbf y) = (X S) \mathbf y$

is exactly a sought for orthogonal projection of $\mathbf y$ onto an image of X (see the picture below and note that as explained in the next section the image of X is just a subspace generated by column vectors of X). A few popular ways to find such a matrix S are described below.

Inverting the matrix of the normal equations

The algebraic solution of the normal equations can be written as

$\hat{\boldsymbol{\beta}} = (\mathbf X^ \top \mathbf X )^{-1} \mathbf X^ \top \mathbf y = \mathbf X^%2B \mathbf y$

where X⁺ is the Moore–Penrose pseudoinverse of X. Although this equation is correct, and can work in many applications, it is not computationally efficient to invert the normal equations matrix. An exception occurs in numerical smoothing and differentiation where an analytical expression is required.

If the matrix X^TX is well-conditioned and positive definite, that is, it has full rank, the normal equations can be solved directly by using the Cholesky decomposition R^TR, where R is an upper triangular matrix, giving:

$R^\top R \hat{\boldsymbol{\beta}} = X^\top \mathbf y.$

The solution is obtained in two stages, a forward substitution step, solving for z:

$R^\top \mathbf z = X^\top \mathbf y,$

followed by a backward substitution, solving for $\hat{\boldsymbol{\beta}}$

$R \hat{\boldsymbol{\beta}}= \mathbf z.$

Both substitutions are facilitated by the triangular nature of R.

See example of linear regression for a worked-out numerical example with three parameters.

Orthogonal decomposition methods

Orthogonal decomposition methods of solving the least squares problem are slower than the normal equations method but are more numerically stable, from not having to form the product X^TX.

The residuals are written in matrix notation as

$\mathbf r= \mathbf y - X \hat{\boldsymbol{\beta}}.$

The matrix X is subjected to an orthogonal decomposition; the QR decomposition will serve to illustrate the process.

$X=QR \$

where Q is an m×m orthogonal matrix and R is an m×n matrix which is partitioned into a n×n upper triangular block, R_n, and a (m − n)×n zero block 0.

$R= \begin{bmatrix} R_n \\ \mathbf{0} \end{bmatrix}.$

The residual vector is left-multiplied by Q^T.

$Q^\top \mathbf r = Q^\top \mathbf y - \left( Q^\top Q \right) R \hat{\boldsymbol{\beta}}= \begin{bmatrix} \left(Q^\top \mathbf y \right)_n - R_n \hat{\boldsymbol{\beta}} \\ \left(Q^\top \mathbf y \right)_{m-n} \end{bmatrix} = \begin{bmatrix} \mathbf u \\ \mathbf v \end{bmatrix}$

Because Q is orthogonal, the sum of squares of the residuals, s, may be written as:

$s = \|\mathbf r \|^2 = \mathbf r^\top \mathbf r = \mathbf r^\top Q Q^\top \mathbf r = \mathbf u^\top \mathbf u %2B \mathbf v^\top \mathbf v$

Since v doesn't depend on β, the minimum value of s is attained when the upper block, u, is zero. Therefore the parameters are found by solving:

$R_n \hat{\boldsymbol{\beta}} =\left(Q^\top \mathbf y \right)_n.$

These equations are easily solved as R_n is upper triangular.

An alternative decomposition of X is the singular value decomposition (SVD)^[1]

$X = U \Sigma V^T. \$

where U is m by m orthogonal matrix, V is n by n orthogonal matrix and $\Sigma$ is an m by n matrix with all its elements outside of the main diagonal equal to 0. The (pseudo)-inverse of $\Sigma$ is easily obtained by inverting its non-zero diagonal elements. Hence,

$\mathbf X V \Sigma^%2B U^T = U \Sigma V^T V \Sigma^%2B U^T = U P U^T,$

where P is obtained from $\Sigma$ by replacing its non-zero diagonal elements with ones. Since X and $\Sigma$ are obviously of the same rank (one of the many advantages of singular value decomposition)

$\mathbf X V \Sigma^%2B U^T = U P U^T$

is an orthogonal projection onto the image (column-space) of X and in accordance with a general approach described in the introduction above,

$\beta = V\Sigma^%2B U^T \mathbf y$

is a solution of a least squares problem. This method is the most computationally intensive, but is particularly useful if the normal equations matrix, X^TX, is very ill-conditioned (i.e. if its condition number multiplied by the machine's relative round-off error is appreciably large). In that case, including the smallest singular values in the inversion merely adds numerical noise to the solution. This can be cured using the truncated SVD approach, giving a more stable and exact answer, by explicitly setting to zero all singular values below a certain threshold and so ignoring them, a process closely related to factor analysis.

Properties of the least-squares estimators

The gradient equations at the minimum can be written as

$(\mathbf y - X \hat{\boldsymbol{\beta}})^\top X=0.$

A geometrical interpretation of these equations is that the vector of residuals, $\mathbf y - X \hat{\boldsymbol{\beta}}$ is orthogonal to the column space of X, since the dot product $(\mathbf y-X\hat{\boldsymbol{\beta}})\cdot X \mathbf v$ is equal to zero for any conformal vector, v. This means that $\mathbf y - X \boldsymbol{\hat \beta}$ is the shortest of all possible vectors $\mathbf{y}- X \boldsymbol \beta$ , that is, the variance of the residuals is the minimum possible. This is illustrated at the right.

Introducing $\hat{\boldsymbol{\gamma}}$ and a matrix K with the assumption that a matrix $[X \ K]$ is non-sigular and K^T X = 0 (cf. Orthogonal projections), the residual vector should satisfy the following equation:

$\hat{\mathbf{r}} \triangleq \mathbf{y} - X \hat{\boldsymbol{\beta}} = K \hat{{\boldsymbol{\gamma}}}.$

The equation and solution of linear least squares are thus described as follows:

$\mathbf{y} = \begin{bmatrix}X & K\end{bmatrix} \begin{pmatrix} \hat{\boldsymbol{\beta}} \\ \hat{\boldsymbol{\gamma}} \end{pmatrix} ,$

$\begin{pmatrix} \hat{\boldsymbol{\beta}} \\ \hat{\boldsymbol{\gamma}} \end{pmatrix} = \begin{bmatrix}X & K\end{bmatrix}^{-1} \mathbf{y} = \begin{bmatrix} (X^\top X)^{-1} X^\top \\ (K^\top K)^{-1} K^\top \end{bmatrix} \mathbf{y} .$

If the experimental errors, $\epsilon \,$ , are uncorrelated, have a mean of zero and a constant variance, $\sigma$ , the Gauss-Markov theorem states that the least-squares estimator, $\hat{\boldsymbol{\beta}}$ , has the minimum variance of all estimators that are linear combinations of the observations. In this sense it is the best, or optimal, estimator of the parameters. Note particularly that this property is independent of the statistical distribution function of the errors. In other words, the distribution function of the errors need not be a normal distribution. However, for some probability distributions, there is no guarantee that the least-squares solution is even possible given the observations; still, in such cases it is the best estimator that is both linear and unbiased.

For example, it is easy to show that the arithmetic mean of a set of measurements of a quantity is the least-squares estimator of the value of that quantity. If the conditions of the Gauss-Markov theorem apply, the arithmetic mean is optimal, whatever the distribution of errors of the measurements might be.

However, in the case that the experimental errors do belong to a normal distribution, the least-squares estimator is also a maximum likelihood estimator.^[2]

These properties underpin the use of the method of least squares for all types of data fitting, even when the assumptions are not strictly valid.

Limitations

An assumption underlying the treatment given above is that the independent variable, x, is free of error. In practice, the errors on the measurements of the independent variable are usually much smaller than the errors on the dependent variable and can therefore be ignored. When this is not the case, total least squares also known as Errors-in-variables models, or Rigorous least squares, should be used. This can be done by adjusting the weighting scheme to take into account errors on both the dependent and independent variables and then following the standard procedure.^[3]^[4]

In some cases the (weighted) normal equations matrix X^TX is ill-conditioned. When fitting polynomials the normal equations matrix is a Vandermonde matrix. Vandermode matrices become increasingly ill-conditioned as the order of the matrix increases. In these cases, the least squares estimate amplifies the measurement noise and may be grossly inaccurate. Various regularization techniques can be applied in such cases, the most common of which is called ridge regression. If further information about the parameters is known, for example, a range of possible values of $\mathbf{\hat{\boldsymbol{\beta}}}$ , then various techniques can be used to increase the stability of the solution. For example, see constrained least squares.

Another drawback of the least squares estimator is the fact that the norm of the residuals, $\| \mathbf y - X\hat{\boldsymbol{\beta}} \|$ is minimized, whereas in some cases one is truly interested in obtaining small error in the parameter $\mathbf{\hat{\boldsymbol{\beta}}}$ , e.g., a small value of $\|\hat{\boldsymbol{\beta}}-\hat{\boldsymbol{\beta}}\|$ . However, since $\hat{\boldsymbol{\beta}}$ is unknown, this quantity cannot be directly minimized. If a prior probability on $\hat{\boldsymbol{\beta}}$ is known, then a Bayes estimator can be used to minimize the mean squared error, $E \left\{ \| \hat{\boldsymbol{\beta}} - \hat{\boldsymbol{\beta}} \|^2 \right\}$ . The least squares method is often applied when no prior is known. Surprisingly, when several parameters are being estimated jointly, better estimators can be constructed, an effect known as Stein's phenomenon. For example, if the measurement error is Gaussian, several estimators are known which dominate, or outperform, the least squares technique; the best known of these is the James–Stein estimator. This is an example of more general shrinkage estimators that have been applied to regression problems.

Weighted linear least squares

Parameter errors, correlation and confidence limits

The estimated parameter values are linear combinations of the observed values

$\hat{\boldsymbol{\beta}} = (X^\top W X)^{-1} X^\top W \mathbf y. \,$

Therefore an expression for the residuals (i.e., the estimated errors in the observations) can be obtained by error propagation from the errors in the observations. Let the variance-covariance matrix for the observations be denoted by M and that of the parameters by M^β. Then,

$M^\beta= (X^\top W X)^{-1} X^\top W M W^\top X (X^\top W X)^{-1}.$

When W = M⁻¹ this simplifies to

$M^\beta=(X^\top W X)^{-1}.$

When unit weights are used (W = I) it is implied that the experimental errors are uncorrelated and all equal: M = σ²I, where σ² is the variance of an observation, and I is the identity matrix. In this case σ² is approximated by $\frac{S}{m-n}$ , where S is the minimum value of the objective function

$M^\beta=\frac{S}{m-n}(X^\top X)^{-1}.$

The denominator, m-n, is the statistical degrees of freedom; see effective degrees of freedom for generalizations for the case of correlated observations. In all cases, the variance of the parameter $\beta_i$ is given by $M^\beta_{ii}$ and the covariance between parameters $\beta_i$ and $\beta_j$ is given by $M^\beta_{ij}$ . Standard deviation is the square root of variance and the correlation coefficient is given by $\rho_{ij} = M^\beta_{ij}/\sigma_i/\sigma_j$ . These error estimates reflect only random errors in the measurements. The true uncertainty in the parameters is larger due to the presence of systematic errors which, by definition, cannot be quantified. Note that even though the observations may be un-correlated, the parameters are always correlated.

It is often assumed, for want of any concrete evidence, that the error on an observation belongs to a normal distribution with a mean of zero and standard deviation $\sigma$ . Under that assumption the following probabilities can be derived.

68% within the interval $\hat \beta \pm \sigma$

95% within the interval $\hat \beta \pm 2\sigma$

99% within the interval $\hat \beta \pm 2.5\sigma$

The assumption is not unreasonable when m >> n. If the experimental errors are normally distributed the parameters will belong to a Student's t-distribution with m − n degrees of freedom. When m >> n Student's t-distribution approximates to a Normal distribution. Note, however, that these confidence limits cannot take systematic error into account. Also, parameter errors should be quoted to one significant figure only, as they are subject to sampling error.^[6]

When the number of observations is relatively small, Chebychev's inequality can be used for an upper bound on probabilities, regardless of any assumptions about the distribution of experimental errors: the maximum probabilities that a parameter will be more than 1, 2 or 3 standard deviations away from its expectation value are 100%, 25% and 11% respectively.

Residual values and correlation

The residuals are related to the observations by

$\mathbf{\hat r} = \mathbf y- X \hat{\boldsymbol{\beta}}= \mathbf y- H \mathbf y = (I - H) \mathbf y$

where H is the symmetric, idempotent matrix known as the hat matrix:

$H = X \left(X^\top W X \right)^{-1}X^\top W$

and I is the identity matrix. The variance-covariance matrice of the residuals, M^r is given by

$M^{\mathbf r}=\left(I-H \right) M \left(I-H \right)^\top.$

Thus the residuals are correlated, even if the observations are not.

When $W = M^{-1}$ ,

$M^{\mathbf r}=\left(I-H \right) M.$

The sum of residual values is equal to zero whenever the model function contains a constant term. Left-multiply the expression for the residuals by X^T:

$X^\top \hat{\mathbf r}=X^\top \mathbf y- X^\top X \hat{\boldsymbol{\beta}} = X^\top \mathbf y- (X^\top X)(X^\top X)^{-1}X^\top \mathbf y= \mathbf 0$

Say, for example, that the first term of the model is a constant, so that $X_{i1}=1$ for all i. In that case it follows that

$\sum_i^m X_{i1} \hat r_i=\sum_i^m \hat r_i=0.$

Thus, in the motivational example, above, the fact that the sum of residual values is equal to zero it is not accidental but is a consequence of the presence of the constant term, α, in the model.

If experimental error follows a normal distribution, then, because of the linear relationship between residuals and observations, so should residuals,^[7] but since the observations are only a sample of the population of all possible observations, the residuals should belong to a Student's t-distribution. Studentized residuals are useful in making a statistical test for an outlier when a particular residual appears to be excessively large.

Objective function

The objective function can be written as

$S=\mathbf y^\top (I-H)^\top (I-H) \mathbf y= \mathbf y^\top (I-H) \mathbf y$

since (I – H) is also symmetric and idempotent. It can be shown from this,^[8] that the expected value of S is m-n. Note, however, that this is true only if the weights have been assigned correctly. If unit weights are assumed, the expected value of S is $(m-n)\sigma^2$ , where $\sigma^2$ is the variance of an observation.

If it is assumed that the residuals belong to a Normal distribution, the objective function, being a sum of weighted squared residuals, will belong to a Chi-squared ( $\chi ^2$ ) distribution with m-n degrees of freedom. Some illustrative percentile values of $\chi ^2$ are given in the following table.^[9]

m-n	$\chi ^2 _{0.50}$	$\chi ^2 _{0.95}$	$\chi ^2 _{0.99}$
10	9.34	18.3	23.2
25	24.3	37.7	44.3
100	99.3	124	136

These values can be used for a statistical criterion as to the goodness-of-fit. When unit weights are used, the numbers should be divided by the variance of an observation.

Constrained linear least squares

Often it is of interest to solve a linear least squares problem with an additional constraint on the solution. With constrained linear least squares, the original equation

$\mathbf {X} \boldsymbol {\beta} = \mathbf {y}$

must be satisfied (in the least squares sense) while also ensuring that some other property of $\boldsymbol {\beta}$ is maintained. There are often special purpose algorithms for solving such problems efficiently. Some examples of constraints are given below:

Equality constrained least squares: the elements of $\boldsymbol {\beta}$ must exactly satisfy $\mathbf {L} \boldsymbol {\beta} = \mathbf {d}$
Regularized least squares: the elements of $\boldsymbol {\beta}$ must satisfy $\| \mathbf {L} \boldsymbol {\beta} - \mathbf {d} \| \le \rho$
Non-negative least squares (NNLS): The vector $\boldsymbol {\beta}$ satisfies the vector inequality $\boldsymbol {\beta} \geq \boldsymbol{0}$ that is defined componentwise --- that is, each component must be either positive or zero.
Box-constrained least squares: The vector $\boldsymbol {\beta}$ satisfies the vector inequalities $\boldsymbol{lb} \leq \boldsymbol{\beta} \leq \boldsymbol{ub}$ , each of which is defined componentwise.
Integer constrained least squares: all elements of $\boldsymbol {\beta}$ must be integer (instead of real numbers).
Real constrained least squares: all elements of $\boldsymbol {\beta}$ must be real (rather than complex numbers).
Phase constrained least squares: all elements of $\boldsymbol {\beta}$ must have the same phase.

When the constraint only applies to some of the variables, the mixed problem may be solved using separable least squares by letting $\mathbf {X} = [\mathbf {X_1} \mathbf {X_2} ]$ and $\mathbf {\beta}^T = [\mathbf {\beta_1}^T \mathbf {\beta_2}^T]$ represent the unconstrained (1) and constrained (2) components. Then substituting the least squares solution for $\mathbf {\beta_1}$ , i.e.

$\hat{\boldsymbol {\beta_1}} = \mathbf {X_1}^%2B (\mathbf {y} - \mathbf {X_2} \boldsymbol {\beta_2})$

back into the original expression gives (following some rearrangement) an equation that can be solved as a purely constrained problem in $\mathbf {\beta_2}$ .

$\mathbf{P} \mathbf {X_2} \boldsymbol {\beta_2} = \mathbf{P}\mathbf {y}$

where $\mathbf{P}:=\mathbf{I}-\mathbf {X_1} \mathbf {X_1}^%2B$ is a projection matrix. Following the constrained estimation of $\hat{\boldsymbol {\beta_2}}$ the vector $\hat{\boldsymbol {\beta_1}}$ is obtained from the expression above.

Typical uses and applications

Polynomial fitting: models are polynomials in an independent variable, x:
- Straight line: $f(x, \boldsymbol \beta)=\beta_1 %2B\beta_2 x$ .^[10]
- Quadratic: $f(x, \boldsymbol \beta)=\beta_1 %2B \beta_2 x %2B\beta_3 x^2$ .
- Cubic, quartic and higher polynomials. For regression with high-order polynomials, the use of orthogonal polynomials is recommended.^[11]
Numerical smoothing and differentiation — this is an application of polynomial fitting.
Multinomials in more than one independent variable, including surface fitting
Curve fitting with B-splines ^[3]
Chemometrics, Calibration curve, Standard addition, Gran plot, analysis of mixtures

Uses in data fitting

The primary application of linear least squares is in data fitting. Given a set of m data points $y_1, y_2,\dots, y_m,$ consisting of experimentally measured values taken at m values $x_1, x_2,\dots, x_m$ of an independent variable ( $x_i$ may be scalar or vector quantities), and given a model function $y=f(x, \boldsymbol \beta),$ with $\boldsymbol \beta = (\beta_1, \beta_2, \dots, \beta_n),$ it is desired to find the parameters $\beta_j$ such that the model function "best" fits the data. In linear least squares, linearity is meant to be with respect to parameters $\beta_j,$ so

$f(x, \boldsymbol \beta) = \sum_{j=1}^{n} \beta_j \phi_j(x).$

Here, the functions $\phi_j$ may be nonlinear with respect to the variable x.

Ideally, the model function fits the data exactly, so

$y_i = f(x_i, \boldsymbol \beta)$

for all $i=1, 2, \dots, m.$ This is usually not possible in practice, as there are more data points than there are parameters to be determined. The approach chosen then is to find the minimal possible value of the sum of squares of the residuals

$r_i(\boldsymbol \beta)= y_i - f(x_i, \boldsymbol \beta),\ (i=1, 2, \dots, m)$

so to minimize the function

$S(\boldsymbol \beta)=\sum_{i=1}^{m}r_i^2(\boldsymbol \beta).$

After substituting for $r_i$ and then for $f$ , this minimization problem becomes the quadratic minimization problem above with

$X_{ij}=\phi_j(x_i),$

and the best fit can be found by solving the normal equations.

Further Discussion

The numerical methods for linear least squares are important because linear regression models are among the most important types of model, both as formal statistical models and for exploration of data-sets. The majority of statistical computer packages contain facilities for regression analysis that make use of linear least squares computations. Hence it is appropriate that considerable effort has been devoted to the task of ensuring that these computations are undertaken efficiently and with due regard to round-off error.

Individual statistical analyses are seldom undertaken in isolation, but rather are part of a sequence of investigatory steps. Some of the topics involved in considering numerical methods for linear least squares relate to this point. Thus important topics can be

Computations where a number of similar, and often nested, models are considered for the same data-set. That is, where models with the same dependent variable but different sets of independent variables are to be considered, for essentially the same set of data-points.
Computations for analyses that occur in a sequence, as the number of data-points increases.
Special considerations for very extensive data-sets.

Fitting of linear models by least squares often, but not always, arise in the context of statistical analysis. It can therefore be important that considerations of computation efficiency for such problems extend to all of the auxiliary quantities required for such analyses, and are not restricted to the formal solution of the linear least squares problem.

Rounding errors

Matrix calculations, like any other, are affected by rounding errors. An early summary of these effects, regarding the choice of computation methods for matrix inversion, was provided by Wilkinson. ^[12]

References

^ Lawson, C. L.; Hanson, R. J. (1974). Solving Least Squares Problems. Englewood Cliffs, NJ: Prentice-Hall. ISBN 0138225850.
^ Margenau, Henry; Murphy, George Moseley (1956). The Mathematics of Physics and Chemistry. Princeton: Van Nostrand.
^ ^a ^b Gans, Peter (1992). Data fitting in the Chemical Sciences. New York: Wiley. ISBN 0471934127.
^ Deming, W. E. (1943). Statistical adjustment of Data. New York: Wiley.
^ This implies that the observations are uncorrelated. If the observations are correlated, the expression $\textstyle S=\sum_k \sum_j r_k W_{kj} r_j\,$ applies. In this case the weight matrix should ideally be equal to the inverse of the variance-covariance matrix of the observations.
^ Mandel, John (1964). The Statistical Analysis of Experimental Data. New York: Interscience.
^ Mardia, K. V.; Kent, J. T.; Bibby, J. M. (1979). Multivariate analysis. New York: Academic Press. ISBN 0124712509.
^ Hamilton, W. C. (1964). Statistics in Physical Science. New York: Ronald Press.
^ Spiegel, Murray R. (1975). Schaum's outline of theory and problems of probability and statistics. New York: McGraw-Hill. ISBN 0585267391.
^ Acton, F. S. (1959). Analysis of Straight-Line Data. New York: Wiley.
^ Guest, P. G. (1961). Numerical Methods of Curve Fitting. Cambridge: Cambridge University Press.
^ Wilkinson, J.H. (1963) "Chapter 3: Matrix Computations", Rounding Errors in Algebraic Processes, London: Her Majesty's Stationery Office (National Physical Laboratory, Notes in Applied Science, No.32)

External links

Theory

Least squares and regression analysis

Computational statistics

Least squares · Linear least squares · Non-linear least squares · Iteratively reweighted least squares

Correlation and dependence

Pearson product-moment correlation · Rank correlation (Spearman's rho, Kendall's tau) · Partial correlation · Confounding variable

Regression analysis

Ordinary least squares · Partial least squares · Total least squares · Ridge regression

Regression as a
statistical model

Linear regression	Simple linear regression · Ordinary least squares · Generalized least squares · Weighted least squares · General linear model

Predictor structure	Polynomial regression · Growth curve · Segmented regression · Local regression

Non-standard	Nonlinear regression · Nonparametric · Semiparametric · Robust · Quantile · Isotonic

Non-normal errors	Generalized linear model · Binomial · Poisson · Logistic